Method for capturing virtual space and electronic device using the same

ABSTRACT

An electronic device including a display; and a controller configured to display a playback screen of virtual reality content on the display, in response to a capture command while displaying the playback screen of the virtual reality content, display a virtual icon on the playback screen, and in response to a touch input applied to the virtual icon, capture an image of a virtual space of the virtual reality content based on a position of a user in the virtual space and corresponding to a touch level of the touch input.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to Korean Patent Application No.10-2017-0043887 filed on Apr. 4, 2017 in Korea, the entire contents ofwhich is hereby incorporated by reference in its entirety.

BACKGROUND OF THE DISCLOSURE 1. Field of the Disclosure

The present disclosure relates to a method of capturing a virtual spacein an electronic device and an electronic device using the same.

2. Background of the Disclosure

A head mounted display (HMD) type electronic device is worn on a head ofa user to present an image directly in front of the user's eyes. Such anelectronic device allows a user to enjoy image contents with a largerimage than a TV or screen, and in particular, display a virtual spacescreen so a user can enjoy a virtual space experience.

Studies on allowing such an electronic device to be used in connectionwith another device such as a smart phone, a laptop computer, anavigation device, a slate PC, a tablet PC, an ultrabook, a wearabledevice or the like have been actively performed. In addition, the userwould like to share or memorize experiences in a virtual world using theelectronic device with other people. However, when merely capturing animage viewed by a user in a virtual world into a 2D or 3D image, thereare limitations in expressing various views and emotions experienced bythe user.

SUMMARY OF THE DISCLOSURE

Accordingly, an object of the present disclosure is to provide anelectronic device for allow a user wearing an electronic device togenerate a capture image containing various views and emotionsexperienced by a user in a virtual world without an additional externalcamera, and a method of capturing a virtual space using the same.

In order to solve the foregoing task, an electronic device according toan embodiment of the present disclosure may include a main body; adisplay unit configured to display a playback screen of virtual realitycontent; and a controller configured to generate a virtual icon forcapturing a virtual space of the virtual reality content when a capturecommand is received while playing the virtual reality content, andcontrol the display unit to display the generated virtual icon on theplay screen. Here, when a touch input is applied to the virtual icon,the controller may recognize a user's position in the virtual space whena touch input is applied to the virtual icon, and generate a captureimage of the virtual space corresponding to a touch level of the touchinput based on the recognized user's position.

Moreover, in order to solve the foregoing task, there is provided amethod of controlling an electronic device, and the method may includedisplaying a playback screen of virtual reality content; receiving acapture command while playing the virtual reality content; generatingand displaying a virtual icon for capturing a virtual space of virtualreality content in response to the capture command; recognizing a user'sposition in the virtual space when a touch input is applied to thevirtual icon; and generating a capture image of the virtual spacecorresponding to a touch level of the touch input based on therecognized user's position.

Further scope of applicability of the present invention will becomeapparent from the detailed description given hereinafter. However, thedetailed description and specific examples, while indicating preferredembodiments of the invention, are given by illustration only, sincevarious changes and modifications within the spirit and scope of theinvention will become apparent to those skilled in the art from thisdetailed description.

BRIEF DESCRIPTION OF THE DRAWINGS

The present disclosure will become more fully understood from thedetailed description given herein below and the accompanying drawings,which are given by illustration only, and thus are not limitative of thepresent disclosure, and wherein:

FIGS. 1A and 1B are conceptual views in which an electronic deviceassociated with the present disclosure is seen from differentdirections;

FIG. 2 is a block diagram illustrating the configuration of anelectronic device associated with the present disclosure;

FIGS. 3A and 3B are conceptual diagrams in which a display configurationof an electronic device associated with the present disclosure is seenfrom different directions in a separate manner;

FIGS. 4A and 4B are views illustrating a configuration in which aplayback screen of virtual reality content is displayed on an electronicdevice associated with the present disclosure;

FIG. 5 is a flow chart illustrating a method of generating a captureimage of a virtual space using a virtual icon in an electronic deviceassociated with the present disclosure;

FIGS. 6A, 6B, 7A, 7B and 7C are exemplary conceptual views associatedwith setting a range of a virtual space to be captured according to atouch level applied to a virtual icon in an electronic device associatedwith the present disclosure;

FIGS. 8A, 8B, 8C and 8D are conceptual views illustrating a method ofdisplaying and controlling a capture image of a virtual space in anelectronic device associated with the present disclosure;

FIGS. 9A, 9B, 9C, 10A, 10B and 10C are exemplary conceptual viewsillustrating a method of generating and displaying a capture image basedon a user's movement range in an electronic device associated with thepresent disclosure;

FIGS. 11A, 11B, 11C, 12A, 12B and 12C are exemplary conceptual viewsillustrating a method for setting a range of a virtual space to becaptured in a different manner around a space or an object interactingwith a user in an electronic device associated with the presentdisclosure;

FIG. 13 is an exemplary conceptual view for setting a range of a virtualspace to be captured based on a set time range in an electronic deviceassociated with the present disclosure;

FIG. 14 is a conceptual view illustrating an example in which a degreeof user's movement is shown in a capture image in an electronic deviceassociated with the present disclosure;

FIGS. 15A, 15B, 15C and 15D are conceptual views illustrating an examplein which a capture image is generated using a user's gaze in anelectronic device associated with the present disclosure; and

FIGS. 16A, 16B, 16C, 16D, 17A, 17B, 17C and 17D are conceptual viewsillustrating examples associated with generating a capture image of avirtual space using an external camera in an electronic deviceassociated with the present disclosure.

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Description will now be given in detail according to the exemplaryembodiments disclosed herein, with reference to the accompanyingdrawings. For the sake of brief description with reference to thedrawings, the same or equivalent components will be provided with thesame reference numbers, and description thereof will not be repeated. Asuffix “module” or “unit” used for constituent elements disclosed in thefollowing description is merely intended for easy description of thespecification, and the suffix itself does not give any special meaningor function. The accompanying drawings are used to help easilyunderstand various technical features and it should be understood thatthe embodiments presented herein are not limited by the accompanyingdrawings. As such, the present disclosure should be construed to extendto any alterations, equivalents and substitutes in addition to thosewhich are particularly set out in the accompanying drawings.

Although the terms first, second, etc. may be used herein to describevarious elements, these elements should not be limited by these terms.The terms are used merely for the purpose to distinguish an element fromthe other element. When an element is referred to as being “connectedwith” another element, the element can be connected with the otherelement or intervening elements may also be present. In contrast, whenan element is referred to as being “directly connected with” anotherelement, there are no intervening elements present.

A singular representation may include a plural representation unless itrepresents a definitely different meaning from the context. Terms suchas “include” or “has” are used herein and should be understood that theyare intended to indicate an existence of several components, functionsor steps, disclosed in the specification, and it is also understood thatgreater or fewer components, functions, or steps may likewise beutilized.

Electronic devices, mobile terminals or external terminals describedherein may be implemented using a variety of different types ofterminals. Examples of such terminals include cellular phones, smartphones, user equipment, laptop computers, digital broadcast terminals,personal digital assistants (PDAs), portable multimedia players (PMPs),navigators, portable computers (PCs), slate PCs, tablet PCs, ultrabooks,wearable devices (for example, smart watches, smart glasses, headmounted displays (HMDs)), and the like.

However, it may be easily understood by those skilled in the art thatthe configuration according to the exemplary embodiments of thisspecification can also be applied to a stationary terminal such as adigital TV, a desktop computer and the like, excluding a case of beingapplicable only to the external devices, mobile terminals or externalterminals.

Furthermore, the term “user” used in embodiments described below refersto a person using an electronic device or a device (e.g., a robot withartificial intelligence) using an electronic device. In addition,embodiments described below will describe technologies associated withthe operations of an electronic device for capturing a virtual spaceduring virtual experiences. Moreover, an electronic device describedbelow may denote a wearable device formed to be wearable on a specificportion (e.g., a head portion) of a user's body.

FIGS. 1A and 1B are conceptual views illustrating an example in which anelectronic device associated with the present disclosure is seen fromdifferent directions. An electronic device 100 associated with thepresent disclosure may include at least one of a wireless communicationunit, an input unit (e.g., user input units 123 a, 123 b, 123 c, amicrophone, etc.), a sensing unit 140, an output unit 150 (e.g., adisplay unit 151, an audio output unit 152), an interface unit 160, amemory, a controller, and a power supply unit. The constituent elementsillustrated in FIGS. 1A and 1B may not be necessarily required toimplement an electronic device, and the electronic device 100 describedin the present specification may have a greater or less number ofelements than those listed above.

Referring to FIG. 1A, the electronic device 100 associated with thepresent disclosure may be formed to be worn on a head portion (or head,face) of a user (human body), and may include a frame portion (a case, ahousing, a cover, etc.) for the same. The frame portion may be formed ofa flexible material to facilitate wearing. It is illustrated in thedrawing that the frame portion includes a first frame 101 and a secondframe 102 with different materials.

For example, the first frame 101 provides a space in which at least oneof the constituent elements described in FIG. 1A can be disposed, andthe second frame 102 supports or fixes the first frame 101 to be mountedon the head portion of the user (human body). The frame portion may bereferred to as a main body (or an electronic device main body) or a body(or an electronic device body).

If a frame portion including the first frame 101 and the second frame102 is regarded as one electronic device main body, then the main bodyof the electronic device associated with the present disclosure may beformed in various shapes. Specifically, the main body may include aplurality of surfaces constituting a preset angle. The plurality ofsurfaces denote surfaces located outside the main body of the electronicdevice 100. From this point of view, the plurality of surfaces maydenote surfaces (an external surface, an outer surface, etc.) of theelectronic device 100. The plurality of surfaces may respectively have aflat or bent shape.

In addition, a wearing portion 112 comes into contact with a portion ofthe user's body. For example, the wearing portion 112 may use a bandformed of an elastic material so the frame portion is brought into closecontact with the eyes of the user's face.

The main body (frame portion) is supported on the head portion toprovide a space for mounting various components. As illustrated in thedrawing, electronic components such as a camera 121, a display unit 151,user input units 123 a, 123 b, 123 c, a controller 180, a sensing unit140, an interface unit 160, and the like may be mounted on the firstframe 101.

An electronic component such as an audio output unit 152 or the like maybe mounted on the second frame 102. However, the present disclosure isnot limited to this, and other constituent elements required for theelectronic device may be disposed in various ways on the first frame 101and the second frame 102 by the user's selection. In other words, theelectronic device 100 described herein may have a greater or less numberof constituent elements than those listed above.

The controller 180 of the electronic device controls various electroniccomponents provided in the electronic device 100. The camera 121 may beprovided in the display unit 151 of the main body. For example, thecamera 121 may be disposed on one surface (for example, a front surface)of the main body of the electronic device 100. The camera 121 can alsobe disposed adjacent to at least one of the left and right eyes, andformed to capture (receive, enter) a front image. The camera 121 mayalso be disposed adjacent to the eyes to face a front side, and thus thecamera 121 may acquire a scene viewed by a user as an image.

FIG. 1A illustrates one camera 121 is provided therein, but the presentdisclosure is not limited to this. That is, the camera 221 can beprovided with a plural number to acquire a stereoscopic image.

The electronic device 100 may include a sensing unit 140. The sensingunit 140 may include a proximity sensor, an illumination sensor, a touchsensor, an acceleration sensor, a magnetic sensor, a gravity sensor(G-sensor), a gyroscope sensor, a gyroscope, a motion sensor, an RGBsensor, an infrared (IR) sensor, a finger scan sensor, an ultrasonicsensor, an optical sensor, a microphone (refer to microphone 122), abattery gauge, an environmental sensor (for example, a barometer, ahygrometer, a thermometer, a radiation detection sensor, a heatdetection sensor, a gas detection sensor, etc.), a chemical sensor (forexample, an electronic nose, a healthcare sensor, a biometric sensor,etc.).

For example, the controller 180 can detect the movement of theelectronic device using a gyroscope sensor, a gravity sensor, a motionsensor, and the like included in the sensing unit 140. In anotherexample, the controller 180 can detect a subject approaching thevicinity of the electronic device body using a proximity sensor, anilluminance sensor, a magnetic sensor, an infrared sensor, an ultrasonicsensor, an optical sensor, or the like included in the sensing unit 140.

The electronic device 100 may be provided with user input units 123 a,123 b, 123 c manipulated to receive a control command. The user inputunits 123 a, 123 b, 123 c can employ any method if it is a tactilemanner allowing the user to perform manipulation with a tactile feelingsuch as touch, push, wheel manipulation or the like. FIGS. 1A and 1Billustrate user input units 123 a, 123 b, 123 c with a touch inputmethod, a touch input method and a wheel manipulation method are allprovided on the frame portion.

Furthermore, the electronic device 100 may include a microphone whichprocesses input sound into electric audio data, and an audio outputmodule 152 for outputting an audible sound. The audio output module 152can transfer the audible sound in a general audio output manner or anosteoconductive manner. When the audio output unit 152 is implemented inthe osteoconductive manner, the audio output unit 152 can be closelyadhered onto the head portion when the user wears the electronic device100 and vibrate the user's skull to transfer audible sounds.

The display unit 151 may be mounted on the frame unit to perform therole of displaying screen information (for example, image, video, etc.)in front of the user's eyes. The display unit 151 may be disposed tocorrespond to at least one of the left and right eyes to display screeninformation in front of the user's eyes when the user wears theelectronic device 100. In other words, the display unit 151 may beformed to cover at least one of the user's left and right eyes (or toface at least one of the user's left and right eyes).

For example, the display unit 151 of an electronic device associatedwith the present disclosure can be located within the electronic devicebody. Specifically, the display unit 151 can be disposed within theelectronic device, and disposed at a position facing the user's eyeswhen the user wears the electronic device on the user's head portion.

Furthermore, the electronic device 100 may include a lens unit 120 toallow the user to view screen information displayed on the display unit151. In other words, the electronic device 100 associated with thepresent disclosure may be formed to transmit all screen information (orlight) displayed on the display unit 151 to the user's eye balls (orfield of view). For example, the lens unit 120 may be disposed tocorrespond to at least one of the user's both eyes (i.e., left and righteyes). Furthermore, the lens unit 120 may be disposed to lie between theuser's eye balls and the display unit 151.

The lens unit 120 may have a varying viewing angle according to adistance between the user's eye balls and the display unit, and thus theposition thereof may be formed to vary by the user's control.Furthermore, the lens unit 120 may be formed through a concave lens, aconvex lens or a combination thereof.

Furthermore, the display unit 151 can project an image to the user'seyes using a prism. Also, the prism can be formed of a transparentmaterial to allow the user to view the projected image and a generalfield of view (a range seen through the user's eyes) at the same time.

As described above, an image displayed through the display unit 151 canbe seen to overlap with a general field of view. The electronic device100 can thus provide augmented reality (AR) in which a virtual imageoverlaps with an image or background of reality using thecharacteristics of the display to show one image. In other words, thedisplay unit 151 may be formed not to allow external light to passtherethrough to implement virtual reality (VR) or formed to allowexternal light to pass therethrough to implement augmented reality (AR).

Alternatively, the electronic device 100 can be functionally connectedto a terminal device 200 to use the terminal device 200 as the displayunit 151 of the electronic device 100. Hereinafter, an example in whichthe terminal device 200 is used as the display unit 151 has beendescribed. When the terminal device 200 is used as the display unit 151of the electronic device 100 as described above, the frame portion ofthe electronic device 100 can be configured to include a predetermineddocking space for the terminal device 200 to be coupled thereto, and inthis instance, they are coupled to each other so the terminal device 200is accommodated into the docking space. When the terminal device 200 isused as the display unit 151 of the electronic device 100 as describedabove, the camera 121 and the sensing unit 140 may be replaced with acamera and a sensing unit provided in the terminal device 200 (refer toFIG. 1A).

In addition, when the electronic device 100 is electrically connected tothe terminal device 200, the controller 180 of the electronic device maycontrol the terminal device 200. When the terminal device 200 is mountedon the electronic device body 100 as described above, a configurationincluding the camera 121, the sensing unit 140, the display unit 151,the controller 180, and the like in the electronic device 100 may bereplaced with a configuration including a camera, a sensing unit, adisplay unit, and a controller included in the terminal device 200.

Although the terminal device 200 is mounted on the main body of theelectronic device 100 to be used as a display unit according to thepresent disclosure, the present disclosure is not limited thereto, andthe present disclosure may of course include when the electronic device100 is provided therein with the display unit 151 to reduce its weight.

The configuration of the electronic device 100 will be described indetail on the assumption that a configuration including the terminaldevice 200 is mounted on the electronic device main body 200, and thecamera 121, the sensing unit 140, the display unit 151, the controller180, and the like in the electronic device 100 is replaced with aconfiguration including the camera, the sensing unit, the display unit,and the controller included in the terminal device 200.

Next, FIG. 2 is a block diagram illustrating a terminal deviceassociated with the present disclosure, and FIGS. 3A and 3B areconceptual diagrams in which a display configuration of an electronicdevice associated with the present disclosure is seen from differentdirections in a separate manner.

The terminal device 200 may include components, such as a wirelesscommunication unit 210, an input unit 220, a sensing unit 240, an outputunit 250, an interface unit 260, a memory 270, a controller 280, a powersupply unit 290 and the like. FIG. 2 illustrates the mobile terminalhaving various components, but implementing all of the illustratedcomponents is not a requirement. Greater or fewer components mayalternatively be implemented.

In more detail, the wireless communication unit 210 of those constituentelements may include one or more modules which permit wirelesscommunications between the terminal device 200 and a wirelesscommunication system, between the terminal device 200 and anotherterminal device 200, or between the terminal device 200 and an externalserver. Furthermore, the wireless communication unit 210 may include oneor more modules for connecting the terminal device 200 to one or morenetworks.

The wireless communication unit 210 may include at least one of abroadcast receiving module 211, a mobile communication module 212, awireless Internet module 213, a short-range communication module 214, alocation information module 215 and the like. The input unit 220 mayinclude a camera 221 for inputting an image signal, a microphone 222 oran audio input module for inputting an audio signal, or a user inputunit 223 (for example, a touch key, a push key (or a mechanical key),etc.) for allowing a user to input information. Audio data or image datacollected by the input unit 220 may be analyzed and processed by auser's control command.

The sensing unit 240 may include at least one sensor which senses atleast one of internal information of the mobile terminal, a surroundingenvironment of the mobile terminal and user information. For example,the sensing unit 240 may include a proximity sensor 241, an illuminationsensor 242, a touch sensor, an acceleration sensor, a magnetic sensor, aG-sensor, a gyroscope sensor, a motion sensor, an RGB sensor, aninfrared (IR) sensor, a finger scan sensor, a ultrasonic sensor, anoptical sensor (for example, refer to the camera 221), a microphone 222,a battery gage, an environment sensor (for example, a barometer, ahygrometer, a thermometer, a radiation detection sensor, a thermalsensor, a gas sensor, etc.), and a chemical sensor (for example, anelectronic nose, a health care sensor, a biometric sensor, etc.).Further, the mobile terminal disclosed herein may utilize information insuch a manner of combining information sensed by at least two sensors ofthose sensors.

The output unit 250 may be configured to output an audio signal, a videosignal or a tactile signal. The output unit 250 may include a displayunit 251, an audio output module 252, a haptic module 253, an opticaloutput module 254 and the like. The display unit 251 may have aninter-layered structure or an integrated structure with a touch sensorso as to implement a touch screen. The touch screen may provide anoutput interface between the terminal device 200 and a user, as well asfunctioning as the user input unit 223 which provides an input interfacebetween the terminal device 200 and the user.

The interface unit 260 may serve as an interface with various types ofexternal devices connected with the terminal device 200. The interfaceunit 260, for example, may include wired or wireless headset ports,external power supply ports, wired or wireless data ports, memory cardports, ports for connecting a device having an identification module,audio input/output (I/O) ports, video I/O ports, earphone ports, or thelike. The terminal device 200 may execute an appropriate controlassociated with a connected external device, in response to the externaldevice being connected to the interface unit 260.

Furthermore, the memory 270 may store a plurality of applicationprograms (or applications) executed in the terminal device 200, data foroperations of the terminal device 200, instruction words, and the like.At least some of those application programs may be downloaded from anexternal server via wireless communication. Some others of thoseapplication programs may be installed within the terminal device 200 atthe time of being shipped for basic functions of the terminal device 200(for example, receiving a call, placing a call, receiving a message,sending a message, etc.). Further, the application programs may bestored in the memory 270, installed in the terminal device 200, andexecuted by the controller 280 to perform an operation (or a function)of the terminal device 200.

The controller 280 may typically control an overall operation of theterminal device 200 in addition to the operations associated with theapplication programs. The controller 280 may provide or processinformation or functions appropriate for a user by processing signals,data, information and the like, which are input or output by theaforementioned components, or activating the application programs storedin the memory 270.

Furthermore, the controller 280 can control at least part of thecomponents illustrated in FIG. 2, in order to drive the applicationprograms stored in the memory 270. In addition, the controller 280 candrive the application programs by combining at least two of thecomponents included in the terminal device 200 for operation.

The power supply unit 290 can receive external power or internal powerand supply appropriate power required for operating respective elementsand components included in the terminal device 200 under the control ofthe controller 280. The power supply unit 290 may include a battery, andthe battery may be an embedded battery or a replaceable battery.

At least part of those elements and components may be combined toimplement operation and control of the mobile terminal or a controlmethod of the mobile terminal according to various exemplary embodimentsdescribed herein. Also, the operation and control or the control methodof the mobile terminal may be implemented in the mobile terminal in sucha manner of activating at least one application program stored in thememory 270.

First, the wireless communication unit 210 will be described. Thebroadcast receiving module 211 of the wireless communication unit 210may receive a broadcast signal and/or broadcast associated informationfrom an external broadcast managing entity via a broadcast channel. Thebroadcast channel may include a satellite channel and a terrestrialchannel. At least two broadcast receiving modules 211 may be provided inthe terminal device 200 to simultaneously receive at least two broadcastchannels or switch the broadcast channels.

The mobile communication module 212 may transmit/receive wirelesssignals to/from at least one of network entities, for example, a basestation, an external mobile terminal, a server, and the like, on amobile communication network, which is constructed according totechnical standards or transmission methods for mobile communications(for example, Global System for Mobile Communication (GSM), CodeDivision Multi Access (CDMA), Wideband CDMA (WCDMA), High Speed DownlinkPacket access (HSDPA), Long Term Evolution (LTE), etc.)

The wireless signals may include audio call signal, video (telephony)call signal, or various formats of data according totransmission/reception of text/multimedia messages. The wirelessInternet module 213 denotes a module for wireless Internet access. Thismodule may be internally or externally coupled to the terminal device200. The wireless Internet module 213 may transmit/receive wirelesssignals via communication networks according to wireless Internettechnologies.

Examples of such wireless Internet access may include Wireless LAN(WLAN), Wireless Fidelity (Wi-Fi) Direct, Digital Living NetworkAlliance (DLNA), Wireless Broadband (Wibro), Worldwide Interoperabilityfor Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA),Long Term Evolution (LTE), and the like. The wireless Internet module213 may transmit/receive data according to at least one wirelessInternet technology within a range including even Internet technologieswhich are not aforementioned.

From the perspective that the wireless Internet accesses according toWibro, HSDPA, GSM, CDMA, WCDMA, LET and the like are executed via amobile communication network, the wireless Internet module 213 whichperforms the wireless Internet access via the mobile communicationnetwork may be understood as a type of the mobile communication module212.

The short-range communication module 214 denotes a module forshort-range communications. Suitable technologies for implementing theshort-range communications may include BLUETOOTH™, Radio FrequencyIDentification (RFID), Infrared Data Association (IrDA), Ultra-WideBand(UWB), ZigBee, Near Field Communication (NFC), Wireless-Fidelity(Wi-Fi), Wi-Fi Direct, and the like. The short-range communicationmodule 214 may support wireless communications between the terminaldevice 200 and a wireless communication system, between the terminaldevice 200 and another terminal device 200, or between the mobileterminal and a network where another terminal device 200 (or an externalserver) is located, via wireless personal area networks.

Here, the another terminal device 200 may be a wearable device, forexample, a smart watch, smart glasses or a head mounted display (HMD),which can exchange data with the terminal device 200 (or to like datawith the terminal device 200). The short-range communication module 214may sense (recognize) a wearable device, which can communicate with themobile terminal), near the terminal device 200. In addition, when thesensed wearable device is a device which is authenticated to communicatewith the terminal device 200 according to the present disclosure, thecontroller 280 may transmit at least part of data processed in theterminal device 200 to the wearable device via the short-rangecommunication module 214. Hence, a user of the wearable device may usethe data processed in the terminal device 200 on the wearable device.For example, when a call is received in the terminal device 200, theuser can answer the call using the wearable device. Also, when a messageis received in the terminal device 200, the user can check the receivedmessage using the wearable device.

The location information module 215 denotes a module for detecting orcalculating a position of the mobile terminal. An example of thelocation information module 215 may include a Global Position System(GPS) module or a Wi-Fi module. For example, when the mobile terminaluses the GPS module, a position of the mobile terminal may be acquiredusing a signal sent from a GPS satellite. As another example, when themobile terminal uses the Wi-Fi module, a position of the mobile terminalmay be acquired based on information related to a wireless access point(AP) which transmits or receives a wireless signal to or from the Wi-Fimodule. According to the need, the location information module 215 mayperform any function of the other modules of the wireless communicationunit 210 to obtain data on the location of the mobile terminal. As amodule used to acquire the location (or current location) of the mobileterminal, the location information module 215 is not limited to a modulefor directly calculating or acquiring the location of the mobileterminal.

Next, the input unit 220 may be configured to provide an audio or videosignal (or information) input to the mobile terminal or informationinput by a user to the mobile terminal. For the input of the audioinformation, the terminal device 200 may include one or a plurality ofcameras 221. The camera 221 may process image frames of still picturesor video obtained by image sensors in a video call mode or a capturemode. The processed image frames may be displayed on the display unit251. Further, the plurality of cameras 221 disposed in the terminaldevice 200 may be arranged in a matrix configuration. By use of thecameras 221 having the matrix configuration, a plurality of imageinformation having various angles or focal points may be input into theterminal device 200. Also, the plurality of cameras 221 may be arrangedin a stereoscopic structure to acquire a left image and a right imagefor implementing a stereoscopic image.

The microphone 222 may process an external audio signal into electricaudio data. The processed audio data may be utilized in various mannersaccording to a function being executed in the terminal device 200 (or anapplication program being executed). Further, the microphone 222 mayinclude assorted noise removing algorithms to remove noise generated inthe course of receiving the external audio signal.

The user input unit 223 may receive information input by a user. Wheninformation is input through the user input unit 223, the controller 280may control an operation of the terminal device 200 to correspond to theinput information. The user input unit 223 may include a mechanicalinput element (or a mechanical key, for example, a button located on afront/rear surface or a side surface of the terminal device 200, a domeswitch, a jog wheel, a jog switch, etc.), and a touch-sensitive inputmeans. As one example, the touch-sensitive input means may be a virtualkey, a soft key or a visual key, which is displayed on a touch screenthrough software processing, or a touch key which is disposed on aportion except for the touch screen. Further, the virtual key or thevisual key may be displayable on the touch screen in various shapes, forexample, graphic, text, icon, video or a combination thereof.

Also, the sensing unit 240 may sense at least one of internalinformation of the mobile terminal, surrounding environment informationof the mobile terminal and user information, and generate a sensingsignal corresponding to it. The controller 280 may control an operationof the terminal device 200 or execute data processing, a function or anoperation associated with an application program installed in the mobileterminal based on the sensing signal. Hereinafter, description will begiven in more detail of representative sensors of various sensors whichmay be included in the sensing unit 240.

First, a proximity sensor 241 refers to a sensor to sense presence orabsence of an object approaching to a surface to be sensed, or an objectdisposed near a surface to be sensed, by using an electromagnetic fieldor infrared rays without a mechanical contact. The proximity sensor 241may be arranged at an inner region of the mobile terminal covered by thetouch screen, or near the touch screen. The proximity sensor 241 mayhave a longer lifespan and a more enhanced utility than a contactsensor.

The proximity sensor 241, for example, may include a transmissive typephotoelectric sensor, a direct reflective type photoelectric sensor, amirror reflective type photoelectric sensor, a high-frequencyoscillation proximity sensor, a capacitance type proximity sensor, amagnetic type proximity sensor, an infrared rays proximity sensor, andso on. When the touch screen is implemented as a capacitance type, theproximity sensor 241 may sense proximity of a pointer to the touchscreen by changes of an electromagnetic field, which is responsive to anapproach of an object with conductivity. In this instance, the touchscreen (touch sensor) may be categorized into a proximity sensor.

For the sake of brief explanation, a state that the pointer ispositioned to be proximate onto the touch screen without contact will bereferred to as “proximity touch,” whereas a state that the pointersubstantially comes in contact with the touch screen will be referred toas “contact touch.” For the position corresponding to the proximitytouch of the pointer on the touch screen, such position will correspondto a position where the pointer faces perpendicular to the touch screenupon the proximity touch of the pointer. The proximity sensor 241 maysense proximity touch, and proximity touch patterns (e.g., distance,direction, speed, time, position, moving state, etc.). Further, thecontroller 280 may process data (or information) corresponding to theproximity touches and the proximity touch patterns sensed by theproximity sensor 241, and output visual information corresponding to theprocess data on the touch screen. In addition, the controller 280 maycontrol the terminal device 200 to execute different operations orprocess different data (or information) according to whether a touchwith respect to the same point on the touch screen is either a proximitytouch or a contact touch.

A touch sensor may sense a touch (or touch input) applied onto the touchscreen (or the display unit 251) using at least one of various types oftouch methods, such as a resistive type, a capacitive type, an infraredtype, a magnetic field type, and the like. As one example, the touchsensor may be configured to convert changes of pressure applied to aspecific part of the display unit 251 or a capacitance occurring from aspecific part of the display unit 251, into electric input signals.Also, the touch sensor may be configured to sense not only a touchedposition and a touched area, but also touch pressure. Here, a touchobject is an object to apply a touch input onto the touch sensor.Examples of the touch object may include a finger, a touch pen, a styluspen, a pointer or the like.

When touch inputs are sensed by the touch sensors as described above,corresponding signals may be transmitted to a touch controller. Thetouch controller may process the received signals, and then transmitcorresponding data to the controller 280. Accordingly, the controller280 may sense which region of the display unit 251 has been touched.Here, the touch controller may be a component separate from thecontroller 280 or the controller 280 itself.

Further, the controller 280 may execute a different control or the samecontrol according to a type of an object which touches the touch screen(or a touch key provided in addition to the touch screen). Whether toexecute the different control or the same control according to theobject which gives a touch input may be decided based on a currentoperating state of the terminal device 200 or a currently executedapplication program.

Meanwhile, the touch sensor and the proximity sensor may be executedindividually or in combination, to sense various types of touches, suchas a short (or tap) touch, a long touch, a multi-touch, a drag touch, aflick touch, a pinch-in touch, a pinch-out touch, a swipe touch, ahovering touch, and the like.

An ultrasonic sensor may be configured to recognize position informationassociated with a sensing object by using ultrasonic waves. Thecontroller 280 may calculate a position of a wave generation sourcebased on information sensed by an illumination sensor and a plurality ofultrasonic sensors. Since light is much faster than ultrasonic waves, atime for which the light reaches the optical sensor may be much shorterthan a time for which the ultrasonic wave reaches the ultrasonic sensor.The position of the wave generation source may be calculated using thefact. In more detail, the position of the wave generation source may becalculated by using a time difference from the time that the ultrasonicwave reaches based on the light as a reference signal.

The camera 221 constructing the input unit 220 may be a type of camerasensor. The camera sensor may include at least one of a photo sensor anda laser sensor. The camera 221 and the laser sensor may be combined todetect a touch of the sensing object with respect to a 3D stereoscopicimage. The photo sensor may be laminated on the display device. Thephoto sensor may be configured to scan a movement of the sensing objectin proximity to the touch screen. In more detail, the photo sensor mayinclude photo diodes and transistors at rows and columns to scan contentplaced on the photo sensor by using an electrical signal which changesaccording to the quantity of applied light. Namely, the photo sensor maycalculate the coordinates of the sensing object according to variationof light to thus obtain position information of the sensing object.

The display unit 251 may output information processed in the terminaldevice 200. For example, the display unit 251 may display executionscreen information of an application program driven in the terminaldevice 200 or user interface (UI) and graphic user interface (GUI)information in response to the execution screen information.

Furthermore, the display unit 251 may also be implemented as astereoscopic display unit for displaying stereoscopic images. Thestereoscopic display unit may employ a stereoscopic display scheme suchas stereoscopic scheme (a glass scheme), an auto-stereoscopic scheme(glassless scheme), a projection scheme (holographic scheme), or thelike.

The audio output unit 252 may output audio data received from thewireless communication unit 210 or stored in the memory 260 in a callsignal reception mode, a call mode, a record mode, a voice recognitionmode, a broadcast reception mode, and the like. Also, the audio outputunit 252 may also provide audible output signals related to a particularfunction (e.g., a call signal reception sound, a message receptionsound, etc.) performed by the terminal device 200. The audio output unit252 may include a receiver, a speaker, a buzzer or the like.

A haptic module 253 may generate various tactile effects that can befelt by a user. A representative example of tactile effect generated bythe haptic module 253 may be vibration. The intensity, pattern and thelike of vibration generated by the haptic module 253 may be controlledby a user's selection or the settings of the controller. For example,the haptic module 253 may output different vibrations in a combinedmanner or in a sequential manner.

Besides vibration, the haptic module 253 may generate various othertactile effects, including an effect by stimulation such as a pinarrangement vertically moving with respect to a contact skin, a sprayforce or suction force of air through a jet orifice or a suctionopening, a touch on the skin, a contact of an electrode, electrostaticforce, etc., an effect by reproducing the sense of cold and warmth usingan element that can absorb or generate heat, and the like.

The haptic module 253 may be implemented to allow the user to feel atactile effect through a muscle sensation such as the user's fingers orarm, as well as transferring the tactile effect through a directcontact. Two or more haptic modules 253 may be provided according to theconfiguration of the terminal device 200.

An optical output unit 254 may output a signal for indicating an eventgeneration using light of a light source. Examples of events generatedin the terminal device 200 may include a message reception, a callsignal reception, a missed call, an alarm, a schedule notice, an emailreception, an information reception through an application, and thelike.

A signal output by the optical output unit 254 may be implemented so themobile terminal emits monochromatic light or light with a plurality ofcolors. The signal output may be terminated as the mobile terminalsenses a user's event checking.

The interface unit 260 may serve as an interface with every externaldevice connected with the terminal device 200. For example, theinterface unit 260 may receive data transmitted from an external device,receive power to transfer to each element within the terminal device200, or transmit internal data of the terminal device 200 to an externaldevice. For example, the interface unit 260 may include wired orwireless headset ports, external power supply ports, wired or wirelessdata ports, memory card ports, ports for connecting a device having anidentification module, audio input/output (I/O) ports, video I/O ports,earphone ports, or the like.

The identification module may be a chip that stores various informationfor authenticating authority of using the terminal device 200 and mayinclude a user identity module (UIM), a subscriber identity module(SIM), a universal subscriber identity module (USIM), and the like. Inaddition, the device having the identification module (referred to as“identifying device’, hereinafter) may take the form of a smart card.Accordingly, the identifying device may be connected with the terminal200 via the interface unit 260.

When the terminal device 200 is connected with an external cradle, theinterface unit 260 may serve as a passage to allow power from the cradleto be supplied to the terminal device 200 therethrough or may serve as apassage to allow various command signals input by the user from thecradle to be transferred to the mobile terminal therethrough. Variouscommand signals or power input from the cradle may operate as signalsfor recognizing that the mobile terminal is properly mounted on thecradle.

The memory 270 may store programs for operations of the controller 280and temporarily store input/output data (for example, phonebook,messages, still images, videos, etc.). The memory 270 may store datarelated to various patterns of vibrations and audio which are output inresponse to touch inputs on the touch screen.

The memory 270 may include at least one type of storage medium includinga Flash memory, a hard disk, a multimedia card micro type, a card-typememory (e.g., SD or DX memory, etc.), a Random Access Memory (RAM), aStatic Random Access Memory (SRAM), a Read-Only Memory (ROM), anElectrically Erasable Programmable Read-Only Memory (EEPROM), aProgrammable Read-Only memory (PROM), a magnetic memory, a magneticdisk, and an optical disk. Also, the terminal device 200 may be operatedin relation to a web storage device that performs the storage functionof the memory 270 over the Internet.

As aforementioned, the controller 280 may typically control the generaloperations of the terminal device 200. For example, the controller 280may set or release a lock state for restricting a user from inputting acontrol command with respect to applications when a state of the mobileterminal meets a preset condition.

Furthermore, the controller 280 may also perform controlling andprocessing associated with voice calls, data communications, videocalls, and the like, or perform pattern recognition processing torecognize a handwriting input or a picture drawing input performed onthe touch screen as characters or images, respectively. In addition, thecontroller 280 may control one or combination of those components inorder to implement various exemplary embodiment disclosed herein on theterminal device 200.

The power supply unit 290 may receive external power or internal powerand supply appropriate power required for operating respective elementsand components included in the terminal device 200 under the control ofthe controller 280. The power supply unit 290 may include a battery. Thebattery may be an embedded battery which is rechargeable or bedetachably coupled to the terminal body for charging.

Furthermore, the power supply unit 290 may include a connection port.The connection port may be configured as one example of the interfaceunit 260 to which an external (re)charger for supplying power torecharge the battery is electrically connected.

As another example, the power supply unit 290 may be configured torecharge the battery in a wireless manner without use of the connectionport. Here, the power supply unit 290 may receive power, transferredfrom an external wireless power transmitter, using at least one of aninductive coupling method which is based on magnetic induction or amagnetic resonance coupling method which is based on electromagneticresonance.

Various embodiments described herein may be implemented in acomputer-readable or its similar medium using, for example, software,hardware, or any combination thereof.

Referring to FIGS. 3A and 3B, the terminal device 200 disclosed hereinmay be provided with a bar-type terminal body. However, the presentdisclosure is not limited to this, but also may be applicable to variousstructures such as watch type, clip type, glasses type or folder type,flip type, slide type, swing type, swivel type, or the like, in whichtwo and more bodies are combined with each other in a relatively movablemanner. The description to be associated with a specific type of mobileterminal or on a specific type of mobile terminal will be also typicallyapplied to another type of mobile terminal.

The terminal device 200 may include a case (casing, housing, cover,etc.) forming the appearance of the terminal. In this embodiment, thecase may be divided into a front case 201 and a rear case 202. Variouselectronic components may be incorporated into a space formed betweenthe front case 201 and the rear case 202. At least one middle case maybe additionally disposed between the front case 201 and the rear case202

A display unit 251 may be disposed on a front surface of the terminalbody to output information. As illustrated, a window 251 a of thedisplay unit 251 may be mounted to the front case 201 so as to form thefront surface of the terminal body together with the front case 201.

In some cases, electronic components may also be mounted to the rearcase 202. Examples of those electronic components mounted to the rearcase 202 may include a detachable battery, an identification module, amemory card and the like. Here, a rear cover 203 for covering theelectronic components mounted may be detachably coupled to the rear case202. Therefore, when the rear cover 203 is detached from the rear case202, the electronic components mounted to the rear case 202 may beexternally exposed.

As illustrated, when the rear cover 203 is coupled to the rear case 202,a side surface of the rear case 202 may be partially exposed. In somecases, upon the coupling, the rear case 202 may also be completelyshielded by the rear cover 203. Further, the rear cover 203 may includean opening for externally exposing a camera 221 b or an audio outputmodule 252 b. The cases 201, 202, 203 may be formed by injection-moldingsynthetic resin or may be formed of a metal, for example, stainlesssteel (STS), titanium (Ti), or the like.

Unlike the example which the plurality of cases form an inner space foraccommodating such various components, the terminal device 200 may beconfigured such that one case forms the inner space. In this example, aterminal device 200 having a uni-body formed so synthetic resin or metalextends from a side surface to a rear surface may also be implemented.

Further, the terminal device 200 may include a waterproofing unit forpreventing an introduction of water into the terminal body. For example,the waterproofing unit may include a waterproofing member which islocated between the window 251 a and the front case 201, between thefront case 201 and the rear case 202, or between the rear case 202 andthe rear cover 203, to hermetically seal an inner space when those casesare coupled.

The terminal device 200 may include a display unit 251, first and secondaudio output modules 252 a, 252 b, a proximity sensor 241, anillumination sensor 252, an optical output module 254, first and secondcameras 221 a, 221 b, first and second manipulation units 223 a, 223 b,a microphone 222, an interface unit 260 and the like.

Hereinafter, description will be given of an exemplary terminal device200 that the display unit 251, the first audio output module 252 a, theproximity sensor 241, the illumination sensor 242, the optical outputmodule 254, the first camera 221 a and the first manipulation unit 223 aare disposed on the front surface of the terminal body, the secondmanipulation unit 223 b, the microphone 222 and the interface unit 260are disposed on a side surface of the terminal body, and the secondaudio output module 252 b and the second camera 221 b are disposed on arear surface of the terminal body, with reference to FIGS. 3A and 3B.

Here, those components may not be limited to the arrangement, but beexcluded or arranged on another surface if necessary. For example, thefirst manipulation unit 223 a may not be disposed on the front surfaceof the terminal body, and the second audio output module 252 b may bedisposed on the side surface other than the rear surface of the terminalbody.

The display unit 251 may output information processed in the terminaldevice 200. For example, the display unit 251 may display executionscreen information of an application program driven in the terminaldevice 200 or user interface (UI) and graphic user interface (GUI)information in response to the execution screen information.

The display unit 251 may include at least one of a liquid crystaldisplay (LCD), a thin film transistor-liquid crystal display (TFT-LCD),an organic light emitting diode (OLED), a flexible display, a3-dimensional (3D) display, and an e-ink display. The display unit 251may be implemented in two or more in number according to a configuredaspect of the terminal device 200. For instance, a plurality of thedisplay units 251 may be arranged on one surface to be spaced apart fromor integrated with each other, or may be arranged on different surfaces.

The display unit 251 may include a touch sensor which senses a touchonto the display unit so as to receive a control command in a touchingmanner. When a touch is input to the display unit 251, the touch sensormay be configured to sense this touch and the controller 280 maygenerate a control command corresponding to the touch. The content whichis input in the touching manner may be a text or numerical value, or amenu item which can be indicated or designated in various modes.

The touch sensor may be configured in a form of film having a touchpattern. The touch sensor may be a metal wire, which is disposed betweenthe window 251 a and a display on a rear surface of the window 251 a orpatterned directly on the rear surface of the window 251 a. Or, thetouch sensor may be integrally formed with the display. For example, thetouch sensor may be disposed on a substrate of the display or within thedisplay.

The display unit 251 may form a touch screen together with the touchsensor. Here, the touch screen may serve as the user input unit 223 (seeFIG. 2A). Therefore, the touch screen may replace at least some offunctions of the first manipulation unit 223 a.

The first audio output module 252 a may be implemented in the form of areceiver for transferring voice sounds to the user's ear or a loudspeaker for outputting various alarm sounds or multimedia reproductionsounds.

The window 251 a of the display unit 251 may include a sound hole foremitting sounds generated from the first audio output module 252 a.Here, the present disclosure may not be limited to this. It may also beconfigured such that the sounds are released along an assembly gapbetween the structural bodies (for example, a gap between the window 251a and the front case 201). In this instance, a hole independently formedto output audio sounds may not be seen or hidden in terms of appearance,thereby further simplifying the appearance of the terminal device 200.

The optical output module 254 may output light for indicating an eventgeneration. Examples of the event generated in the terminal device 200may include a message reception, a call signal reception, a missed call,an alarm, a schedule notice, an email reception, information receptionthrough an application, and the like. When a user's event checking issensed, the controller may control the optical output unit 254 to stopthe output of the light.

The first camera 221 a may process video frames such as still or movingimages obtained by the image sensor in a video call mode or a capturemode. The processed video frames may be displayed on the display unit251 or stored in the memory 270.

The first and second manipulation units 223 a, 223 b are examples of theuser input unit 223, which may be manipulated by a user to input acommand for controlling the operation of the terminal device 200. Thefirst and second manipulation units 223 a, 223 b may also be commonlyreferred to as a manipulating portion, and may employ any method if itis a tactile manner allowing the user to perform manipulation with atactile feeling such as touch, push, scroll or the like.

The drawings are illustrated on the basis that the first manipulationunit 223 a is a touch key, but the present disclosure is not limited tothis. For example, the first manipulation unit 223 a may be configuredwith a mechanical key, or a combination of a touch key and a push key.

The content received by the first and second manipulation units 223 a,223 b may be set in various ways. For example, the first manipulationunit 223 a may be used by the user to input a command such as menu, homekey, cancel, search, or the like, and the second manipulation unit 223 bmay be used by the user to input a command, such as controlling a volumelevel being output from the first or second audio output module 252 a or252 b, switching into a touch recognition mode of the display unit 251,or the like.

Further, as another example of the user input unit 223, a rear inputunit may be disposed on the rear surface of the terminal body. The rearinput unit may be manipulated by a user to input a command forcontrolling an operation of the terminal device 200. The content inputmay be set in various ways. For example, the rear input unit may be usedby the user to input a command, such as power on/off, start, end, scrollor the like, controlling a volume level being output from the first orsecond audio output module 252 a or 252 b, switching into a touchrecognition mode of the display unit 251, or the like. The rear inputunit may be implemented into a form allowing a touch input, a push inputor a combination thereof.

The rear input unit may be disposed to overlap the display unit 251 ofthe front surface in a thickness direction of the terminal body. As oneexample, the rear input unit may be disposed on an upper end portion ofthe rear surface of the terminal body such that a user can easilymanipulate it using a forefinger when the user grabs the terminal bodywith one hand. However, the present disclosure may not be limited tothis, and the position of the rear input unit may be changeable.

When the rear input unit is disposed on the rear surface of the terminalbody, a new user interface may be implemented using the rear input unit.Also, the aforementioned touch screen or the rear input unit maysubstitute for at least part of functions of the first manipulation unit223 a located on the front surface of the terminal body. Accordingly,when the first manipulation unit 223 a is not disposed on the frontsurface of the terminal body, the display unit 251 may be implemented tohave a larger screen.

Further, the terminal device 200 may include a finger scan sensor whichscans a user's fingerprint. The controller may use fingerprintinformation sensed by the finger scan sensor as an authentication means.The finger scan sensor may be installed in the display unit 251 or theuser input unit 223.

The microphone 222 may be formed to receive the user's voice, othersounds, and the like. The microphone 222 may be provided at a pluralityof places, and configured to receive stereo sounds.

The interface unit 260 may serve as a path allowing the terminal device200 to exchange data with external devices. For example, the interfaceunit 260 may be at least one of a connection terminal for connecting toanother device (for example, an earphone, an external speaker, or thelike), a port for near field communication (for example, an InfraredData Association (IrDA) port, a Bluetooth port, a wireless LAN port, andthe like), or a power supply terminal for supplying power to theterminal device 200. The interface unit 260 may be implemented in theform of a socket for accommodating an external card, such as SubscriberIdentification Module (SIM), User Identity Module (UIM), or a memorycard for information storage.

The second camera 221 b may be further mounted to the rear surface ofthe terminal body. The second camera 221 b may have an image capturingdirection, which is substantially opposite to the direction of the firstcamera unit 221 a.

The second camera 221 b may include a plurality of lenses arranged alongat least one line. The plurality of lenses may also be arranged in amatrix configuration. The cameras may be referred to as an “arraycamera.” When the second camera 221 b is implemented as the arraycamera, images may be captured in various manners using the plurality oflenses and images with better qualities may be obtained. A flash 224 maybe disposed adjacent to the second camera 221 b. When an image of asubject is captured with the camera 221 b, the flash 224 may illuminatethe subject.

The second audio output module 252 b may further be disposed on theterminal body. The second audio output module 252 b may implementstereophonic sound functions in conjunction with the first audio outputmodule 252 a, and may be also used for implementing a speaker phone modefor call communication.

At least one antenna for wireless communication may be disposed on theterminal body. The antenna may be installed in the terminal body orformed on the case. For example, an antenna which configures a part ofthe broadcast receiving module 211 (refer to FIG. 2) may be retractableinto the terminal body. Alternatively, an antenna may be formed in aform of film to be attached onto an inner surface of the rear cover 203or a case including a conductive material may serve as an antenna.

A power supply unit 290 (refer to FIG. 2) for supplying power to theterminal device 200 may be disposed on the terminal body. The powersupply unit 290 may include a batter 291 which is mounted in theterminal body or detachably coupled to an outside of the terminal body.

The battery 291 may receive power via a power source cable connected tothe interface unit 260. Also, the battery 291 may be (re)chargeable in awireless manner using a wireless charger. The wireless charging may beimplemented by magnetic induction or electromagnetic resonance.

Further, the drawing illustrates that the rear cover 203 is coupled tothe rear case 202 for shielding the battery 291, so as to preventseparation of the battery 291 and protect the battery 291 from anexternal impact or foreign materials. When the battery 291 is detachablefrom the terminal body, the rear case 203 may be detachably coupled tothe rear case 202.

An accessory for protecting an appearance or assisting or extending thefunctions of the terminal device 200 may further be provided on theterminal device 200. As one example of the accessory, a cover or pouchfor covering or accommodating at least one surface of the terminaldevice 200 may be provided. The cover or pouch may cooperate with thedisplay unit 251 to extend the function of the terminal device 200.Another example of the accessory may be a touch pen for assisting orextending a touch input onto a touch screen.

Hereinafter, FIGS. 4A and 4B are views illustrating a configuration inwhich a playback screen of virtual reality content is displayed on theelectronic device 100 associated with the present disclosure. Theelectronic device 100 according to the present disclosure or theterminal device 200 mounted on the electronic device 100 can providevirtual reality content through the display unit 151. The virtualreality content exhibits virtual reality that is different from actualreality. In other words, as illustrated in FIG. 4A, the virtual realitycontent provides a three-dimensional spatial image 400 for a userwearing the electronic device 100 to be mistaken as if he or she is in aspecific area or another space while preventing actual reality frombeing seen. Accordingly, the user can feel as if he or she is in avirtual space. Such a virtual space can be achieved by providing athree-dimensional spatial image 400 through a left eye image 401 and aright eye image 402, respectively, corresponding to the user's binocularparallax, as illustrated in FIG. 4B.

Such a virtual space or experience in the virtual space may only be feltby a user who wears the electronic device 100. Further, sometimes a userwho wears the electronic device 100 wants to share a virtual space orexperience experienced by the user with another person or to remind theuser himself or herself. In this instance, even if a virtual space iscaptured, it is difficult to transmit the sense of space and experienceexperienced by the user in the virtual space since it is stored as atwo-dimensional image. Otherwise, it is difficult to know specificallywhat the user experienced in the virtual space by capturing only a firstperson perspective that the user himself sees.

As a result, when a capture command is received while a playback screenof virtual reality content is displayed on the display unit 151, theelectronic device 100 according to an embodiment of the presentdisclosure displays a virtual icon for capturing a virtual space of thevirtual reality content on the playback screen. Then, when a touch inputis applied to the displayed virtual icon, the controller 180 of theelectronic device 100 recognizes the user's position in the virtualspace. At this time, the user's position corresponds to the position ofthe user's virtual avatar experiencing a virtual world. Furthermore, thecontroller 180 can generate a capture image of the virtual space tocorrespond to a touch level of the touch input applied to the virtualicon based on the recognized user's position.

Hereinafter, FIG. 5 is a flow chart illustrating a method of generatinga capture image of a virtual space using a virtual icon in theelectronic device associated with the present disclosure. Referring toFIG. 5, a playback screen of virtual reality content is displayedthrough the display unit 151 of the electronic device 100 (S10). Here,the virtual reality content may be one that is stored in the electronicdevice 100 or stored in the terminal device 200 mounted on theelectronic device 100. There is no limitation on the type of the virtualreality content. For example, it may include a home screen or a specificapplication displayed on the display unit 151 of the electronic device100, namely, all cases where all types of programs that can be driven onthe electronic device 100, for example, web browser, video playback,schedule management, call, game, music, document work, message, finance,e-book, traffic information, and the like are displayed in a virtualspace as a three-dimensional spatial image.

When a capture command is generated while the virtual reality content isbeing played back, the controller 180 can recognize the capture command(S20). Such a capture command may be generated in response to variousinput methods or specific event occurrences. Specifically, it includesvarious input methods such as a specific hand gesture, a voice command,a user's gaze or a push input to a physical key, a touch input, and thelike. Alternatively, when there is a specific interval/time/motionchange or the like in the virtual reality content, it may be consideredthat an event corresponding to a trigger signal of the capture commandhas occurred. When such a capture command is generated, the controller180 can determine a time point at which the capture command is generatedor a time point preceding a predetermined time period/interval from atime point at which the capture command is generated. In the lattercase, for example, three-dimensional images that have been stored forabout one minute prior to generating the capture command can beextracted from the buffer to generate one capture image or a pluralityof capture images selected based on a motion change of an object withina three-dimensional image.

Next, in response to the capture command, the controller 180 cangenerate a virtual icon for capturing a virtual space of the virtualreality content, and control the display unit 151 to display thegenerated virtual icon on the playback screen (S30). Here, the virtualicon may be displayed in a shape corresponding to the virtual realitycontent or displayed in a predetermined region (e.g., an upper rightside or a lower right side of the three-dimensional spatial image). Forexample, as illustrated in FIG. 6A, a virtual icon of in a shape of aphotographing icon image 610 may appear on the three-dimensional image601.

In addition, the controller 180 can recognize a touch input applied tothe virtual icon (S40). The touch input to the virtual icon may beperformed through the user's virtual avatar (e.g., a character whoexperiences a virtual world). Alternatively, it may be performed byallowing the user to placing a finger at a point where the virtual iconis actually located in a three-dimensional image. In addition, thecontroller 180 can recognize a touch level of the touch input (e.g., atouch intensity, a touch time, a number of touches, etc.) applied to thevirtual icon.

In response to the touch input applied to the virtual icon as describedabove, the controller 180 can obtain the user's current position in thevirtual space being experienced (or a current position of the user'svirtual avatar). Such a current position of the user can be expressed ina three-dimensional coordinate in the virtual space.

Furthermore, the controller 180 can generate a capture image of thevirtual space corresponding to the touch level of the touch input to thevirtual icon with respect to the recognized the user's position (S50).In other words, the virtual space in all directions including the left,right, upper, and lower sides may be captured as one three-dimensionalimage with the three-dimensional coordinate of the user in the virtualspace as an original point. In other words, a capture image may begenerated using a space that is wider than the three-dimensional imagecurrently seen by the user, the user can be expressed closer to thesense of space and experience that the user felt in a virtual world.

In addition, a virtual world seen from the user viewpoint can be checkedas a 360-degree image through the generated capture image. Moreover, aswill be described in more detail below, it is also be possible to checka virtual world including the user at a third person perspective.

Hereinafter, various examples associated with setting a range of avirtual space to be captured in a different manner according to a touchlevel of a touch applied to a virtual icon generated in response to acapture command will be described with reference to FIGS. 6A, 6B, 7A,7B, and 7C. As illustrated in FIG. 6A, in response to the foregoingcapture command, a virtual icon 610 having a photographing icon imageshape can be displayed in a region of a three-dimensional virtual image601 corresponding to virtual reality content, for example, an upperright side thereof. The virtual icon 610 can appear as a two-dimensionalimage or a three-dimensional image on the three-dimensional virtualimage 601 being played back.

In addition, when a touch level of the touch input applied to thevirtual icon 610 varies, the controller 180 can reduce or extend a rangeof the virtual space to be captured. For example, as a pressure of thetouch input applied to the virtual icon 610 increases or a touch time ofthe touch input increases, a range of the virtual space to be capturedcan be further extended. On the contrary, when a pressure of the touchinput applied to the virtual icon 610 slightly varies or a number oftouches increases, a range of the virtual space to be captured can bereduced to be close to the current position of the user.

Furthermore, according to an example, a graphic object 611 correspondingto a range of the virtual space to be captured may be displayed on thevirtual icon 610. Here, the graphic object may include all various typesof images that determine visual characteristics such as the shape,color, size, and the like of the virtual icon 610 as well as an image(e.g., color, specific object, etc.), a text or the like displayed onthe virtual icon 610. In this instance, when a range of the virtualspace to be captured varies, an image of the graphic object displayed onthe virtual icon 610 is changed to correspond to the changed capturerange.

Specifically, when an input is applied to the virtual icon 610 throughthe user's virtual avatar, the user's actual hand or the like, an imagecan be changed according to a touch level of the touch input. Forexample, when the virtual icon 610 is displayed as a two-dimensionalimage, an image such as color, text (e.g., 2D or 3D display),transparency, and the like of the virtual icon 610 can be changedaccording to a touch level of the touch input, thereby allowing the userto visually recognize a change in the range of the space to be captured.In another example, when the virtual icon 610 is displayed as athree-dimensional image, the 3D depth sense of the virtual icon may bechanged or a visual change such as a three-dimensional movement such asrotation or the like may be performed according to a touch level of thetouch input, thereby visually checking a change in the range of thespace to be captured according to the touch level.

In addition, according to an example, the controller 180 is operated togenerate a capture image at the time when the touch input applied to thevirtual icon 610 is released. In other words, the controller 180 candetermine a range of the virtual space to be captured around the user'sposition at the time when the touch input applied to the virtual icon isreleased. Accordingly, when a capture image is generated at the timewhen the touch input is applied to the virtual icon, it is possible toprevent a time gap from occurring between a three-dimensional spatialimage close to the user's current position and a three-dimensionalspatial image far from the user's current position within a range of thespace to be captured.

FIG. 7A is an example of when a touch less than a predetermined pressureis applied to and then released from the virtual icon 610, wherein athree-dimensional virtual image 701 is captured as a two-dimensionalimage seen from the user position 710. Further, information fornotifying the capture of a three-dimensional image, for example, “2D”text, can be displayed on the virtual icon 610 when the touch input isreleased.

Furthermore, FIGS. 7B and 7C illustrate an example of when a touchexceeding a predetermined pressure is applied to the virtual icon 610,and it is seen that a three-dimensional virtual space around the userlocation 710 is captured while the display of the virtual icon 610changes from “2D” to “3D”, for example. As the pressure of the touchinput increases, a range of the capture space range 621 illustrated inFIG. 7B can be extended as that of the capture space 621 illustrated inFIG. 7C. As described above, the user can recognize the extent of aspatial range to be captured through a visual change of the graphicobject displayed on the virtual icon 610, and the expansion/reduction ofthe range.

Alternatively, according to another example, as a touch time of thetouch input applied to the virtual icon 610 increases, a range of thevirtual space to be captured is extended. Further, since the capturedimage is stored in a 3D format, a capture image corresponding to FIG. 7Cis provided as a screen zoomed out from a capture image corresponding toFIG. 7B.

When a capture image is generated but there is no additional input tothe virtual icon 610 within a predetermined time, the display of thevirtual icon 610 may disappear. In addition, when a plurality of captureimages are generated as an additional input is applied to the virtualicon 610, information (e.g., a number of captured images) indicatingthat a plurality of capture images are generated in the virtual icon 610or another region of the playback screen may be displayed.

The captured image can be stored in the memory 170 or 270 in the form ofa three-dimensional image. In this regard, FIGS. 8A, 8B, 8C and 8Dillustrate various examples associated with a method of displaying andcontrolling the stored capture image of a virtual space.

First, it is seen a folder 812 in which a capture image is storedthrough the display unit 151 of the electronic device 100 or through athumbnail list 801 of images corresponding to the execution of a galleryapplication installed in the terminal device 200 connected to theelectronic device 100 as illustrated in FIG. 8A. In the folder 812 inwhich the capture image is stored, a graphic object 811 indicating thatthe classified images are three-dimensional images, for example, a text“3D’, may be displayed. When the folder 812 in which the capture imageis stored is selected, three-dimensional capture images are displayed inthe order in which they are stored.

The capture image 812′ displayed on the display unit 251 corresponds toa viewpoint and a view angle initially viewed from the user position ina virtual space. It corresponds to position A in FIG. 8D. Furthermore,since the displayed capture image 812′ is a three-dimensional image, athree-dimensional image having a different view angle (e.g., 820) can bedisplayed according to a touch gesture 821 of up, down, left, right orthe like as illustrated in FIG. 8B.

In addition, a viewpoint of the capture image continuously maintains aviewpoint at which the user wearing the electronic device 100 views in avirtual space, namely, a first-person perspective. Specifically, whilethe capture image is being displayed, the controller 180 can control thedisplay unit to change and display a view angle of the capture imagewhile maintaining the first-person perspective of the capture imagebased on a touch gesture applied to the display unit, for example, adrag touch input such as up, down, left and right. Accordingly, the useror a third person can check a virtual world seen from the user positionin a virtual space as in a 360-degree camera, while moving his or hergaze over the entire 360-degree range. It corresponds to position B inFIG. 8D.

In addition, in case a conventional 2D image, when a touch gesture in aleft or right direction is applied to the display unit 251, anext/previous image stored therein is displayed, but herein, differentview angle images of the same 3D capture image are displayed.Consequently, a virtual button (e.g., “←” and/or “→”) for moving to anext/previous 3D capture image can be provided in one region (e.g., anupper right end) of a currently displayed capture image 812′ in FIG. 8B.Furthermore, four direction keys can be provided to guide a view anglethat is viewable on the currently displayed capture image 812′. Inaddition, each direction key of the four direction keys can be displayedas a different image depending on the extent of viewable angle of view.Here, the different image denotes a different visual effect such aslength, thickness, color, blinking or the like of the direction key.

In addition, according to an example, the controller can control thedisplay unit to display a capture image in which a viewpoint is changedwhile the capture image is zoomed out based on a second touch gesture,for example, a pinch-in gesture, applied to the display unit while thecapture image is displayed on the display unit.

Specifically, as illustrated in FIG. 8C, when a pinch-in gesture 822 isadditionally applied to the display unit when a capture image from afirst-person perspective cannot be zoomed out any more, the captureimage from a first-person perspective is changed to a capture image froma second-person or third-person perspective. As a result, as the user ora third person moves backward beyond the user position in a virtualspace as an observation camera, a virtual avatar 830 of himself orherself appears on the capture image. It corresponds to looking at theuser and the virtual space at position C in FIG. 8D.

In addition, a viewpoint modified capture image can be zoomed out usingthe pinch-in gesture 822 until a boundary line of the captured virtualspace is reached. In this instance, a size of the virtual avatar 830 isalso reduced in response to the screen zoom-out. Conversely, when apinch-out gesture is successively performed while a capture image from asecond-person or third-person perspective is displayed, the captureimage may be switched back to the first-person perspective as thecapture image is gradually zoomed in. As a result, the user's virtualavatar naturally disappears.

Further, when a playback command is received while the capture image812′ is displayed, the capture image can rotate in a preset direction sothe entire view angle seen from the first-person perspective can bechecked. In addition, the rotation direction and the rotation speed canbe varied based on a specific object (e.g., a moving object, aninteractive object) included in the capture image. For example, therotation speed can be reduced at a view angle where a specific objectexists or increased at a view angle where the specific object does notexist.

In addition, according to an example, when a viewpoint of the captureimage is changed from a first-person perspective to a second-personperspective or a third-person perspective, a pop-up window askingwhether or not to display the virtual avatar may be displayed.Accordingly, the user can reconsider whether or not to change theviewpoint of the capture image.

In the above, examples associated with setting a virtual range to becaptured based on a touch level of a touch input applied to a virtualicon have been described. Hereinafter, examples associated with settinga range of the virtual space to be captured in connection with theextent of the user's movement in a virtual space will be described.

First, FIGS. 9A, 9B, 9C, 10A, 10B and 10C are exemplary conceptual viewsillustrating a method of generating and displaying a capture image basedon a user's movement range in a virtual space. A user wearing theelectronic device 100 may move in a virtual space according to the typeof virtual reality content. In addition, it is difficult for animpression experienced by the user in a virtual world to be transmittedto the other person only by showing a space viewed by the user in avirtual space in 2D or 3D. Accordingly, it is preferable that a captureimage is generated so other persons can feel various emotions (e.g.,thrill, trembling, height sense, pace, surprise, fear) experienced bythe user while moving in a virtual space.

Thus, according to the present disclosure, while the user's own avatarmoving in a virtual space is displayed on the capture image, a rangethat has been moved by the user can be set to a range of the virtualspace to be captured. In other words, the capture image can be generatedto express the user's mobility and activity in a virtual space.

Specifically, when a touch input is applied to a virtual icon, thecontroller 180 of the electronic device 100 can detect the movementinformation of a virtual avatar moving in a virtual space. Furthermore,a range of the virtual space to be captured can be set based on thedetected movement information. Specifically, the controller 180 can seta range of the virtual space to be captured in a different manner inconsideration of the extent of movement of the virtual avatar in avirtual space, for example, a moving direction and speed thereof, from atime point when the touch input is applied to the virtual icon to amoment when the touch input is released.

Furthermore, a range of the virtual space to be captured may not be setaround the center of a virtual space range in which the user's positionrecognized at the time when the touch input applied to the virtual iconis released is captured. In other words, the position of the virtualavatar does not become the center of the virtual space to be captured.For this purpose, the controller 180 can determine that the position ofthe virtual avatar at the time when the touch input applied to thevirtual icon is released corresponds to the detected movementinformation.

For example, as illustrated in FIG. 9A, when skydiving is performed as avirtual experience, the user's own avatar moves rapidly from top tobottom, and thus a virtual avatar 921 included in a capture image 931 ispositioned below the center coordinate in consideration of the user'smoving direction. In other words, a vertically long 3D rectangular shapeextending upward from the current user's position 820 is set to a VRspace range 911 to be captured in consideration of a spatial range movedby the user in a virtual space 901. In addition, even when a virtualparachute is deployed or the speed of free fall is slowed down at thetime of release of the touch input applied to the virtual icon, a rangeof the VR space to be captured by tracing back a user activity range fora predetermined previous time may be set in various ways.

Furthermore, the visual effect of a panning technique may be displayedon a background image excluding the virtual avatar 921 to show the speedsense of free fall in the capture image 931. For example, as illustratedin FIG. 14, in the capture image, an image of the virtual avatar 1410can be displayed clearly, but a graphic effect that expresses speedsense calculated based on the movement information of the virtualavatar, namely, a panning technique, can be applied the surroundingbackground image.

Here, the panning technique refers to a photographing technique in whicha moving object is stopped but the background is flowing to make thetone dim, thereby showing the feeling of motion. In a virtual space, nomatter how fast the user moves, it is not captured by an optical methodof an actual camera, and thus the background image may be clearlydisplayed as much as he or she likes using the stored 3D data. However,such a photographing technique may be intentionally applied to a captureimage, thereby further relating to the user's sense of movement andactivity in a virtual space.

Further, when using the movement information of the virtual avatar (oranother moving object) in applying the panning technique, the movementinformation of an object existing in a virtual space or a velocity ofthe user himself or herself may be acquired based on the 3D data. Inanother example, the user's own movement speed can be calculated basedon a first still image in the capture image. FIG. 9B is a viewillustrating this.

FIG. 9B illustrates an example in which motion generated in athree-dimensional space is projected onto an image plane. As illustratedin the drawing, when a motion vR is generated in a real space, thismotion is observed as a motion vector vI on the image plane. Accordingto the proposed method, a virtual optical axis is drawn from vI toestimate a size of vR, thereby calculating a total flow velocity in theimage.

In addition, according to an example, when there exist a plurality ofvirtual avatars, for example, when a plurality of persons perform askydiving virtual experience at the same time with reference to FIG. 9A,the controller 180 can intentionally remove one or more of his or herown avatar and the other person's avatar from the capture image togenerate a three-dimensional image. For this purpose, when a range ofthe virtual space to be captured is determined, the controller 180 cangenerate a plurality of capture images in which a plurality of avatarsare removed one by one to provide a plurality of virtual avatarsincluded in the capture image in a selectable manner at the time offinal storage. Then, some avatars can be intentionally removed based onthe user input, and an image of the position where the avatar is removedcan be restored by combining the generated plurality of capture imagesto store only one capture image desired by the user.

Further, a range of the virtual space to be captured is determined basedon movement information when the user himself or herself moves in avirtual space in FIG. 9A, but a capture range is set based on themovement information of a moving object when the user has boarded themoving object (e.g., a virtual bird, a virtual airplane, a virtualroller coaster, etc.).

FIG. 9C illustrates an example in which a viewpoint of the capture imageis changed by touching a virtual avatar included in the capture image.As illustrated in FIG. 9C, when a capture range of the virtual space isset based on the movement information of a virtual avatar, and when thevirtual avatar 921 included in the capture image 931 displayed on thescreen is touched, a viewpoint of the capture image is changed to aviewpoint of the virtual space seen by the virtual avatar, namely, afirst-person perspective (941). The virtual avatar is not displayed onthe capture image 941 whose viewpoint has been changed. Furthermore, aview angle of the 360-degree virtual space from a first-personperspective may be adjusted to a desired direction by using fourdirection keys displayed on the capture image 941 whose viewpoint hasbeen changed or a touch gesture. When a pinch-in gesture is continuouslyperformed on the capture image 941 whose viewpoint has been changed, thevirtual avatar 921 can be displayed while the viewpoint is switched to athird-person perspective as before.

FIGS. 10A, 10B and 10C illustrate an example in which a range of thevirtual space to be captured is set based on a user's movement pathwhile a touch input is applied to a virtual icon 610 (FIG. 6A). Here, itis illustrated an example in which the user captures a space in whichthe user himself or herself is active in a virtual space using a touchinput to the foregoing virtual icon. Referring to FIG. 10A, thecontroller 180 of the electronic device 100 can set the user's position(P1) recognized at the time when a touch input is applied to a virtualicon to one side boundary line of the virtual space to be captured.

Then, the user's position (P2) recognized at the moment of releasing atouch object (e.g., a hand of the user or the virtual avatar) that hasapplied a touch to the virtual icon is set as the other side boundaryline of the virtual space to be captured. Furthermore, a height of avirtual space range to be captured is set in consideration of a heightin which the user's virtual avatar 1020 moves in an up-down directionfrom the position (P1) to the position (P2). In FIG. 10A, a height of avirtual space range to be captured is set based on a height of thevirtual avatar 1020 when the user moves on a flat ground. However, whenthe virtual avatar has jumped or has passed an uphill/downhill roadwhile moving from the position (P1) to the position (P2), the height ofthe virtual space range to be captured may be set different by adding adistance moved in a vertical direction.

The resultant capture image is generated when a touch applied to thevirtual icon is released. In addition, the controller 180 can generate acapture image so a moving path of the virtual avatar is displayed.Accordingly, information 1011 indicating a path the user has moved inthe captured virtual space may be displayed on the generated captureimage.

For example, as illustrated in FIGS. 10A and 10B, the path can bedisplayed in the form of a connected line on the bottom of the captureimage or expressed in the form of a footprint corresponding to acharacter of the virtual avatar. Thus, the path that the user has movedis displayed on the capture image, thereby allowing a third person torecognize the user's activity path in a predetermined section with onlyone capture image.

Furthermore, according to another example, the controller 180 canrecognize the user's position (P2) at the time when a subsequent touchinput is applied to the virtual icon to finally determine a range of thevirtual space to be captured. In this instance, it is not required tocontinuously apply a touch to the virtual icon while the virtual avatar1020 moves from the position (P1) to the position (P2).

Furthermore, the capture image including the virtual avatar 1020 isgenerated, and thus when the virtual avatar 1020 included in the captureimage is touched as illustrated in FIG. 10B, an image 1031 in a virtualworld viewed from the position (P2) by the virtual avatar 1020 isdisplayed as illustrated in FIG. 10C. In this instance, the virtualavatar 1020 disappears from the capture image since it is switched froma third person perspective to a first person perspective of the userhimself or herself.

In addition, according to another example, the controller 180 cangenerate a virtual space corresponding to the virtual avatar's movementpath and movement range as a capture image based on the virtual avatar'smovement path detected within a predetermined time after a touch inputapplied to the virtual icon is released. For example, even in case wherethe virtual avatar 1020 is in the position (P1) or a touch applied tothe virtual icon is released immediately thereafter in FIG. 10A, whenthe virtual avatar has moved beyond a threshold distance or above athreshold speed for a predetermined time (e.g., 3-10 seconds), it ispossible to generate a capture image including a point (e.g., P2) atwhich the virtual avatar is located at a moment when a predeterminedtime has elapsed. According to this, even when the user enters a capturecommand ahead of an actual capturing time, it is possible to put boththe sense of activity and the sense of space of the user into thecapture image. Accordingly, the user does not miss a climax impressionof VR experience due to entering a capture command.

In the above, examples in which the user intentionally sets a range of avirtual space to be captured based on the extent of the user's ownactivity or a touch level to the virtual icon have been described.Hereinafter, examples in which a range of the virtual space to becaptured is set in consideration of the interest of the other person tobe provided with a capture image without allowing the user himself orherself to be centered will be described.

In this regard, FIGS. 11A, 11B, 11C, 12A, 12B and 12C illustrateexamples associated with setting a range of the virtual space to becaptured around a space or object interacting with a user in a virtualworld. First, FIGS. 11A, 11B and 11C are exemplary operations associatedwith capturing a specific sub-region in a range of the virtual space toprovide it to a third person when the virtual space includes a pluralityof partitionable sub-regions.

When a virtual world to which a user experiencing VR experience belongsis indoor, it can include a plurality of partitionable sub-regions (orpartitions). For example, for a virtual model house, room 1, room 2,bathroom, living room, and the like may be the above-mentionedsub-regions. The sub-regions can be determined based on the existenceand size of an object dividing a space such as a wall, a fence, a door,and the like.

FIG. 11A illustrates a capture image generated using a sub-region towhich a user belongs. In FIG. 11A, the sub-region to which the userbelongs is displayed as a three-dimensional image 1101. For thispurpose, when the virtual space includes a plurality of sub-regions, thecontroller 180 can select one sub-region based on the user's positionrecognized at the time when a touch input is applied to the virtualicon, and set the entire selected sub-region to a range of the virtualspace to be captured.

However, according to another example, a specific space can be selectedthrough a user input, setting or the like to generate it as a captureimage even though it is not a sub-region to which the user belongs in avirtual world. For example, even when the user himself or herself may bein room 2 of the virtual model house, it is possible to generate aliving room space as a capture image.

When a specific object (P1) included in a three-dimensional image 1101displayed in FIG. 11A is touched, the selected specific object (e.g., achair) is enlarged and displayed as illustrated in FIG. 11B. Since theenlarged object image 1111 is also a three-dimensional image, it ispossible to rotate the image 1111 through a touch gesture to check theselected object as an image 1112 having another view angle. In addition,a pinch-in/pinch-out touch gesture can be applied to the enlarged objectimage 1111 to check a reduced or enlarged image of the object.

When a specific point (P2) in a space other than the object is touchedin the three-dimensional image 1101 displayed in FIG. 11A, athree-dimensional image 1121 of a sub-region viewed from thecorresponding point (P2) as illustrated in FIG. C is displayed. Here, itis also be possible to change the point of looking at the sub-regionthrough a touch gesture such as up, down, left, right or the like, orcheck a reduced or enlarged image of the sub-region using apinch-in/pinch-out touch gesture.

The user can perform a specified specific gesture or select a virtualkey generated in the enlarged object image 1111 or the three-dimensionalimage 1121 to return at once to the image 1101 of the entire sub-regionillustrated in FIG. 11A.

Furthermore, according to another example, a capture image including avirtual avatar existing in a sub-region may be generated. For a specificexample, when a “conference room” or a “meeting room” in a conference iscaptured as a sub-region, a plurality of virtual avatars performing aconference may be included together in the capture image. In thisinstance, when a specific avatar is touched on the capture image, ameeting room space viewed by the corresponding avatar may be displayed,thereby feeling the atmosphere of the meeting room that the person hasseen.

As described above, a sub-region itself may be provided as a captureimage to allow a desired portion of the sub-region to be viewed orenlarged at a desired angle, thereby having an advantage capable ofsatisfying a variety of different interests of a large number of peoplesharing the same capture image.

Next, FIGS. 12A, 12B and 12C are examples associated with allowing auser to generate a capture image including an interacting object in avirtual space. Specifically, the controller 180 can set a range of avirtual space to be captured, including a virtual object interactingwith an avatar of the user who is experiencing. Here, the virtual objectinteracting with the virtual avatar includes objects that affect theuser himself or herself during a VR experience, affect each other, orare of interest to the user. For example, the virtual object may includea target of the virtual avatar, an avatar of another player who performsa VR experience together with the virtual avatar, an object in which thevirtual avatar is interested, and the like.

For example, as illustrated in FIG. 12A, it is possible to set a range1210 of a virtual space to be captured, including an avatar 1202 ofanother player who performs a VR experience together as well as anavatar 1201 of the user himself or herself. In addition, when there area plurality of players who perform a VR experience together, it may bepossible to add a human condition such as a player who is currentlychatting, a player of the same team, or a number of players locatedclose to the user's current position to limit a range of the virtualspace to be captured.

In another example, as illustrated in FIG. 12B, a capture range 1220 canbe set including an enemy or target 1203 that the user's own avatar 1201is currently fighting. In this instance, a capture image can begenerated so a view from a third-person perspective together with acoordinate of the avatar 1201 and a coordinate of the enemy or target1203 form triangular vertices to exhibit a confrontation between theavatar 1201 and the enemy or target 1203 in a capture image.

In another example, as illustrated in FIG. 12C, the user himself orherself 1201 can set a capture range 1230 including an object 1204looked at with interest during a VR experience. Thus, when the user'sgaze is fixed to a specific object for more than a reference period oftime during a VR experience, the specific object to which the gaze isfixed may be implemented to perform a predetermined reaction (e.g.,sound, movement, etc.). Then, at the moment when a touch input appliedto the virtual icon is released, a capture image including an avatar ofthe user himself or herself and the corresponding object is generated.

Further, when a touch input is applied to a virtual icon for generatinga capture image, a graphic object indicating a virtual object to beincluded in the capture image may be displayed. For example, asdescribed above, it is possible to recognize that a user is capturedtogether by allowing a virtual object to make a specific response orchanging an image of a virtual object (e.g., color change, visual effectoutput to a frame, etc.).

In addition, when a touch input is applied to the graphic objectdisplayed on the virtual object within a reference period of time, arange of the virtual space to be captured is changed. For example, thevirtual object to which the touch input is applied can be removed to seta capture range. Then, the capture image will not include the virtualobject.

Next, FIG. 13 illustrates an example in which a virtual space is zoomedout based on a specific time range to generate a capture image in anelectronic device associated with the present disclosure. For thispurpose, prior to generating a capture image, time information may bereceived through the user input units 123 a, 123 b, 123 c in theelectronic device 100 or a user input unit of the terminal device 200electrically connected to the electronic device 100. For example, from10 a.m. to 2 p.m. may be entered as time information. The entered timeinformation is stored in the memory 170 or the like.

Next, the controller 180 generates a capture image (hereinafter,referred to as “first capture image’) of the virtual space correspondingto a touch level of the touch input applied to the virtual icon. Then,at the moment when a time corresponding to the entered time informationhas elapsed, a capture image (hereinafter, referred to as a “secondcapture image’) of the virtual space in which the user travels for aperiod of time corresponding to the entered time information isgenerated. Assuming that the user moves within the virtual space, thesecond capture image becomes an upper spatial image or zoom-out image ofthe first capture image.

FIG. 13 illustrates a range 1310 of a virtual space corresponding to thesecond capture image around a current position of the avatar 1311. Inthe second capture image, time information 1320 corresponding to acapture range may be displayed. When there is a capture image, namely,the first capture image, generated during the displayed time, a userposition (R1) corresponding to the first capture image may be displayedon the second capture image.

In addition, according to an example, the set time information can bechanged to change a capture range in a different manner. Assuming thatthe longer the set time range the more the user travels, a visual effectsuch as screen zoom-in or screen zoom-out can actually be applied to thecapture image. Alternatively, according to another example, a virtualspace can be captured for each specific time zone to generate aplurality of capture images, and connect the generated capture images soas to store and track the moving path of the avatar.

FIGS. 15A, 15B, 15C and 15D are examples associated with differentlygenerating an initial view of a capture image using a user's gaze. Inother words, the capture image according to the present disclosure canallow a change of the check, enlargement, reduction, and viewpoint ofthe 360-degree view angle, how to set an initial viewing of the captureimage is not so important. However, in some cases, the user may want toemphasize what he or she has viewed during a VR experience as shown inhis eyes.

In a real environment, when an object near the user's position isviewed, the vicinity of the relevant object is not well seen, and whenviewing an object farther away, the vicinity of the object comes intoview more widely. Similarly, even for a lens, it exhibits anout-focusing effect in which a peripheral portion is blurred when asubject is close to the lens, and a pan-focusing effect in which thebackground and the subject are both clearly photographed as the subjectmoves away. Since a VR space generates a capture image usingthree-dimensional data, a capture image can be generated by sharpeningthe object and the background regardless of whether an object viewed bythe user is close to or far from the user's position. However, when theuser wants to emphasize what he or she has seen during a VR experienceas shown in his or her eyes, effects similar to the out-focusing andpan-focusing techniques described above may be applied to the captureimage.

First, referring to FIG. 15A, an out-focusing effect can be applied to acapture image when it is captured while the user views a hand (P1) nearthe user's position in a three-dimensional image 1501 in a virtualworld. Thus, a degree of blur is strongly set such that a hand shape1511 is filled at the center line of the user's gaze, and the backgroundaround the hand shape is blurred. Accordingly, as illustrated in FIG.15B, an initial viewing of the capture image exhibits an out-focusingphenomenon in which the object (hand) viewed by the user is filled onthe screen and the periphery thereof is graphically processed in ablurred manner.

Referring to FIG. 15C, when a capture image is captured while looking ata tree (P2) far from the user's position in the three-dimensional image1501 in a virtual world, a pan-focusing effect can be applied to thecapture image. Specifically, the center line of the user's gaze includesnot only a tree shape 1512 but also other objects in the vicinitythereof, and a degree of blur is slightly set such that the backgrounddistant from the center line is slightly blurred or in a similar manner.As a result, as illustrated in FIG. 15D, an initial viewing of thecapture image exhibits a pan-focusing in which both the object (tree)and the surrounding background come into the screen, and the backgroundaround the tree is graphically processed in a clear manner.

Next, FIGS. 16A, 16B, 16C, 16D, 17A, 17B, 17C and 17D are examplesassociated with generating a capture image of a virtual space using anexternal camera. FIG. 16A illustrates a case where the electronic device100 is electrically connected to the external terminal 300 to receivevirtual reality content from the external terminal 300. It is assumedthat a user wearing the electronic device 100 turns his or her back onthe external terminal 300 to fight with a virtual enemy 1611. Inaddition, in order to generate a capture image using the camera 321 ofthe external terminal 300 such that a face of the user's virtual avatarappears, the position of the user's reality should be changed.

Accordingly, when a playback of the virtual reality content is stoppedto change the position of the user's reality, all the virtual enemies1611 disappear. Thereafter, the user holds an appropriate photographingposition play the virtual reality content back again. In addition, it isrecognized that the user's reality position has been changed to askwhether a game position is changed to an actual position, namely,whether re-sync is performed based on a current position, prior torestarting the VR game again as illustrated in FIG. 16B (1620).Alternatively, according to another example, when the position orposture is changed after the user stops the playback of the content,re-sync may be automatically performed based on a current position for apredetermined period of time.

When the user responds “Yes” to the re-sync request, the position of thegame space is modified to match the changed user's current position.Accordingly, the position of the virtual enemy 1611′ also appears beforethe changed user's current position as illustrated in FIG. 16C. On thecontrary, when the user responds “No” to the re-sync request, thechanged reality is not reflected to play the game according to theuser's position prior to the stop of the play. As a result, asillustrated in FIG. 16D, the user stands in a direction opposite to thecam 321 of the external terminal 300, and the virtual enemy 1611 appearsbehind the user's back based on the previous user's position.

In another example, FIGS. 17A through 17D illustrate a series ofoperations in which a third person captures a user in a virtual worldfrom a third-person perspective with another terminal device. Here, itis assumed that the electronic device 100 and the terminal device of thethird person perform wireless communication with each other. It is alsoassumed that the user wearing the electronic device 100 is set to allowphotographing a virtual world experienced by himself or herself from theoutside.

In this instance, being when the electronic device 100 can transmit avirtual space to an electronic device in the vicinity of the electronicdevice 100, for example, a terminal device of a third person can bebroadcast or advertised through BLE or the like at a predeterminedcycle. In addition, as illustrated in FIG. 17A, as a camera applicationis executed in the terminal device 400 responding to a BLE signal, whena preview screen 1701 is displayed, a virtual key 1710 for executing amixed reality (MR) image view is generated on the relevant previewscreen 1701.

When the third person selects the relevant key 1710, a message (e.g., aguidance prompt, a notification icon, etc.) indicating that the thirdperson desires to photograph himself or herself is generated to the userwearing the electronic device 100. In addition, the message includesinformation associated with a third person. While the user wearing theelectronic device 100 responds, an indicator 1720 indicating that a MRdata is being requested is displayed on the preview screen 1701.

When the user wearing the electronic device 100 accepts photographing,an MR preview screen 1720 synthesized with virtual 3D data is displayedas illustrated in FIG. 17C. At the same time, though not shown in thedrawing, the user wearing the electronic device 100, is informed of whois photographing his or her virtual world, i.e., information on the userof the terminal device 400 (e.g., “Tom is photographing”).

Then, when a third person touches a photograph button 1720 asillustrated in FIG. 17D, a capture image in which the user wearing theelectronic device 100 is mixed with a virtual world and reality isgenerated. The generated capture image is also transmitted to the userwearing the electronic device 100, and appears in virtual reality.

In addition, the capture image is stored at the same time in theelectronic device 100 as well as the terminal device 400 of the thirdperson. In addition, the electronic device 100 may also store a viewimage from a first-person perspective viewed by the user wearing theelectronic device 100 at the time of generating the capture image.

Further, when the user wearing the electronic device 100 views a captureimage stored through a gallery or the like, detailed informationassociated with the capture image is displayed. The detailed informationmay include additional information associated with the characteristicsof mixed reality (MR), such as position information of a virtual world,information of a third person to be photographed, load information, andthe like. In addition, when the position information of the virtualworld is selected, the user can directly enter a virtual world at theposition.

As described above, an electronic device according to an embodiment ofthe present disclosure can intentionally generate a capture image with awider space range including a user himself or herself even when avirtual space is captured from a first-person perspective. Furthermore,a capture image of a first-person perspective can be changed to athird-person perspective or a capture image of a third-personperspective can be changed to a first-person perspective, therebychecking a virtual world from a desired viewpoint and view angle. Inaddition, various visual effects can be applied to a capture image inconsideration of activities such as user's motion in a virtual world,fixation of gaze, and the like, thereby obtaining a capture image almostthe same as that seen by the user in the virtual world.

The foregoing present disclosure may be implemented as codes readable bya computer on a medium written by the program. The computer-readablemedia includes all types of recording devices in which data readable bya computer system can be stored. Examples of the computer-readable mediamay include ROM, RAM, CD-ROM, magnetic tape, floppy disk, and opticaldata storage device, and the like, and also include a device implementedin the form of a carrier wave (for example, transmission via theInternet). The computer may include the controller 180 of the mobileterminal 100. The foregoing embodiments are merely exemplary and are notto be considered as limiting the present disclosure. The scope of theinvention should be determined by reasonable interpretation of theappended claims and all changes that come within the equivalent scope ofthe invention are included in the scope of the invention.

What is claimed is:
 1. An electronic device, comprising: a display; anda controller configured to: display a playback screen of virtual realitycontent on the display, in response to a capture command whiledisplaying the playback screen of the virtual reality content, display avirtual icon on the playback screen, and in response to a touch inputapplied to the virtual icon, capture an image of a virtual space of thevirtual reality content based on a position of a user in the virtualspace and corresponding to a touch level of the touch input.
 2. Theelectronic device of claim 1, wherein in response to the touch inputapplied to the virtual icon being released, the controller is furtherconfigured to determine a range of the virtual space to be captured. 3.The electronic device of claim 2, wherein the controller is furtherconfigured to vary the range of the virtual space to be captured inresponse to a varying touch level of the touch input applied to thevirtual icon.
 4. The electronic device of claim 3, wherein thecontroller is further configured to display a graphic objectcorresponding to the range of the virtual space to be captured on thevirtual icon.
 5. The electronic device of claim 4, wherein thecontroller is further configured change an image of the graphic objectto correspond to the range of the virtual space.
 6. The electronicdevice of claim 1, wherein the captured image comprises a virtual avatarof the user in the virtual space.
 7. The electronic device of claim 6,wherein in response to the touch input applied to the virtual icon, thecontroller is further configured to set a range of the virtual space tobe captured based on movement information of the virtual avatar.
 8. Theelectronic device of claim 7, wherein the movement information comprisesa movement path and movement range of the virtual avatar, and whereinthe controller is further configured to determine whether the movementrange of the virtual avatar within a predetermined time subsequent toreleasing the touch input exceeds a predetermined range, and capture theimage of the virtual space corresponding to the movement path andmovement range of the virtual avatar according to the determination. 9.The electronic device of claim 7, wherein the controller is furtherconfigured to display a graphic image indicating a pace calculated basedon the movement information adjacent to the virtual avatar.
 10. Theelectronic device of claim 7, wherein in response to a touch applied tothe virtual avatar, convert a capture image at a first viewpoint inwhich the virtual avatar is viewed by a third person into a captureimage at a second viewpoint viewed by the virtual avatar.
 11. Theelectronic device of claim 1, wherein in response to a touch applied toa virtual object included in the captured image, the controller isfurther configured to display an enlarged image of the virtual object,and change a view angle of the virtual object based on a drag touchinput applied to the enlarged image.
 12. The electronic device of claim1, wherein a viewpoint of the capture image corresponds to a viewpointat which the user views the virtual space in all directions, and whereinthe controller is further configured to change and display a view angleof the captured image while maintaining the viewpoint of the captureimage based on a first touch gesture applied to the displayed image. 13.The electronic device of claim 12, wherein the controller is furtherconfigured to: display the captured image in which the viewpoint ischanged while the capture image is zoomed out based on a second touchgesture applied to the displayed image, and display the virtual avataron the capture image in which the viewpoint is changed.
 14. Theelectronic device of claim 1, wherein the virtual space includes aplurality of sub-regions, and wherein the controller is furtherconfigured to set a sub-region selected based on the position recognizedat the time when the touch input is applied to the virtual icon as arange of the virtual space to be captured.
 15. The electronic device ofclaim 6, wherein the controller is further configured to set a range ofthe virtual space to be captured by including a virtual objectinteracting with the virtual avatar.
 16. The electronic device of claim15, wherein in response to the touch input applied to the virtual icon,the controller is further configured to display a graphic objectindicating the virtual object to be included in the capture image, andchange a range of the virtual space to be captured based on an inputapplied to the graphic object within a reference time.
 17. Theelectronic device of claim 1, further comprising: wherein the controlleris further configured to: capture a first image of the virtual spacebased on the position of the user, capture a second image including thevirtual space in which the user travels for a period of time included inreceived time information, and display the position of the user in thefirst image in the second image.
 18. A method of controlling anelectronic device, the method comprising: displaying a playback screenof virtual reality content on a display of the electronic device; inresponse to a capture command while displaying the playback screen ofthe virtual reality content, displaying a virtual icon on the playbackscreen; and in response to a touch input applied to the virtual icon,capturing, via a controller, an image of a virtual space of the virtualreality content based on a position of a user in the virtual space andcorresponding to a touch level of the touch input.
 19. The method ofclaim 18, further comprising: in response to the touch input applied tothe virtual icon being released, determining a range of the virtualspace to be captured; and varying the range of the virtual space to becaptured in response to a varying touch level of the touch input appliedto the virtual icon.
 20. The method of claim 19, further comprising:displaying a graphic object corresponding to the range of the virtualspace to be captured on the virtual icon; and changing an image of thegraphic object to correspond to the range of the virtual space.